关于 Linux中系统调优的一些笔记

我突然又明白,死亡是聪明的兄长,我们可以放心地把自己托付给他,他会知道在我们有所准备的适当时刻前来。我也突然懂得,原来痛苦、失望和悲愁不是为了惹恼我们,使我们气馁或者无地自容;它们的存在,是为了使我们心智成熟,臻于完善。——–—赫尔曼·黑塞《彼得·卡门青》

写在前面


  • 推送的的邮件里看到有大佬讲的公共课,听了之后这里整理学习笔记。
  • 因为是公开课,所以讲的很浅,没接触过,这里做为了解,长长见识。
  • 博文内容包括:
内容
系统调优原理概述
如何检测系统的性能瓶颈
如何进行内核参数调优
如何限制服务的资源占用
自定义tuned调优配置集

我突然又明白,死亡是聪明的兄长,我们可以放心地把自己托付给他,他会知道在我们有所准备的适当时刻前来。我也突然懂得,原来痛苦、失望和悲愁不是为了惹恼我们,使我们气馁或者无地自容;它们的存在,是为了使我们心智成熟,臻于完善。——–—赫尔曼·黑塞《彼得·卡门青》


零、系统调优原理概述:

通过调整系统设置来提高计算机资源的利用率数据吞吐量用戶体验的过程,消除或减轻系统中的排队现象

分析系统性能的方法:

分析利用率、饱和、错误 (Utilization, Saturation and Errors (USE))和系统资源
资源 Resource 服务器中的硬件资源(CPU、内存、网络、 磁盘和控制器)
利用率 Utilization 资源提供服务的平均时间(70%)
饱和度 Saturation 资源承担额外工作量的能力,这类工作通常 排队(队列长度)
错误 Errors 错误事件数量

执行系统变更的步骤:

执行系统变更的步骤
设定基线:在调优中,执行性能测试,收集指标
执行变更:在系统上执行变更,建议查阅相应官方文档
验证变更有效性:执行变更后,重新运行测试,比较一些关键指标
取消变更并与基线对比:如有需要,可以取消变更并返回到基线状态
应用最终更改:应用并记录最终变更

一、如何检测系统的性能瓶颈

1、检测系统当前运行负载:vmstat、mpstat、iostat、top 等

装包,一般不需要

1
yum install -y sysstat

sysstat包含监测系统性能及效率的一组工具,这些工具对于我们收集系统性能数据,比如 CPU 使用率、硬盘和网络吞吐数据,这些数据的收集和分析,有利于我们判断系统是否正常运行,是提高系统运行效率、安全运行服务器的得力助手。

vmstat 内存子系统监控

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
┌──[root@liruilongs.github.io]-[~]
└─$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 3228884 2076 594152 0 0 10 6 102 152 0 0 100 0 0
┌──[root@liruilongs.github.io]-[~]
└─$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 3228852 2076 594184 0 0 10 6 102 152 0 0 100 0 0 # 平均值
0 0 0 3228852 2076 594184 0 0 0 0 197 289 0 1 99 0 0
0 0 0 3228852 2076 594184 0 0 0 0 192 290 0 0 100 0 0
1 0 0 3228852 2076 594184 0 0 0 0 180 285 0 0 100 0 0
0 0 0 3228852 2076 594184 0 0 0 0 174 276 1 0 99 0 0
0 0 0 3228852 2076 594184 0 0 0 0 173 267 0 0 100 0 0
0 0 0 3228852 2076 594184 0 0 0 0 176 268 0 0 100 0 0
0 0 0 3228852 2076 594184 0 0 0 0 173 274 0 0 100 0 0
^C
┌──[root@liruilongs.github.io]-[~]
└─$
列名 描述
r 表示运行队列(就是说多少个进程真的分配到CPU),我测试的服务器目前CPU比较空闲,没什么程序在跑,当这个值超过了CPU数目,就会出现CPU瓶颈了。这个也和top的负载有关系,一般负载超过了3就比较高,超过了5就高,超过了10就不正常了,服务器的状态很危险。top的负载类似每秒的运行队列。如果运行队列过大,表示你的CPU很繁忙,一般会造成CPU使用率很高。
b 表示阻塞的进程,这个不多说,进程阻塞,大家懂的。
swpd 虚拟内存已使用的大小,如果大于0,表示你的机器物理内存不足了,如果不是程序内存泄露的原因,那么你该升级内存了或者把耗内存的任务迁移到其他机器。
free 空闲的物理内存的大小。
buff Linux/Unix系统是用来存储,目录里面有什么内容,权限等的缓存.
cache cache直接用来记忆我们打开的文件,给文件做缓冲,我本机大概占用300多M(这里是Linux/Unix的聪明之处,把空闲的物理内存的一部分拿来做文件和目录的缓存,是为了提高 程序执行的性能,当程序使用内存时,buffer/cached会很快地被使用。)
si 每秒从磁盘读入虚拟内存的大小,如果这个值大于0,表示物理内存不够用或者内存泄露了,要查找耗内存进程解决掉。我的机器内存充裕,一切正常。
so 每秒虚拟内存写入磁盘的大小,如果这个值大于0,同上。
bi 块设备每秒接收的块数量,这里的块设备是指系统上所有的磁盘和其他块设备,默认块大小是1024byte,我本机上没什么IO操作,所以一直是0,但是我曾在处理拷贝大量数据(2-3T)的机器上看过可以达到140000/s,磁盘写入速度差不多140M每秒
bo 块设备每秒发送的块数量,例如我们读取文件,bo就要大于0。bi和bo一般都要接近0,不然就是IO过于频繁,需要调整。
in 每秒CPU的中断次数,包括时间中断
cs 每秒上下文切换次数,例如我们调用系统函数,就要进行上下文切换,线程的切换,也要进程上下文切换,这个值要越小越好,太大了,要考虑调低线程或者进程的数目,例如在apache和nginx这种web服务器中,我们一般做性能测试时会进行几千并发甚至几万并发的测试,选择web服务器的进程可以由进程或者线程的峰值一直下调,压测,直到cs到一个比较小的值,这个进程和线程数就是比较合适的值了。系统调用也是,每次调用系统函数,我们的代码就会进入内核空间,导致上下文切换,这个是很耗资源,也要尽量避免频繁调用系统函数。上下文切换次数过多表示你的CPU大部分浪费在上下文切换,导致CPU干正经事的时间少了,CPU没有充分利用,是不可取的。
us 用户CPU时间,我曾经在一个做加密解密很频繁的服务器上,可以看到us接近100,r运行队列达到80(机器在做压力测试,性能表现不佳)。
sy CPU运行kernel代码的时间,比如执行系统调用,系统CPU时间,如果太高,表示系统调用时间长,例如是IO操作频繁。
id CPU处于idle状态的时间,一般来说,id + us + sy = 100,一般我认为id是空闲CPU使用率,us是用户CPU使用率,sy是系统CPU使用率。
wa CPU处于idle状态,因为所有正在运行的进程都在等待I/O操作完成,因此当前无可以调度的进程。
st CPU花费在执行系统上运行的虚拟机的时间

mpstat CUP子系统监控

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
┌──[root@liruilongs.github.io]-[~]
└─$ mpstat 1
Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/23/2021 _x86_64_ (2 CPU)

01:41:22 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
01:41:23 AM all 0.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.48
01:41:24 AM all 0.00 0.00 0.51 0.00 0.00 0.00 0.00 0.00 0.00 99.49
01:41:25 AM all 0.00 0.00 0.00 0.00 0.00 0.52 0.00 0.00 0.00 99.48
01:41:26 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
01:41:27 AM all 0.00 0.00 0.52 0.00 0.00 0.00 0.00 0.00 0.00 99.48
01:41:28 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
01:41:29 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
^C
Average: all 0.07 0.00 0.15 0.00 0.00 0.07 0.00 0.00 0.00 99.71
┌──[root@liruilongs.github.io]-[~]
└─$ mpstat 1 3
Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/23/2021 _x86_64_ (2 CPU)

01:42:06 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
01:42:07 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
01:42:08 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
01:42:09 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
┌──[root@liruilongs.github.io]-[~]
└─$

iostat IO子系统监控

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
┌──[root@liruilongs.github.io]-[~]
└─$ iostat 1 3
Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/23/2021 _x86_64_ (2 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
0.21 0.00 0.24 0.05 0.00 99.50

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 0.56 19.86 12.15 387999 237243

avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 0.52 0.00 99.48

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 1.00 8.00 0.00 8 0

avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 0.00 0.00 100.00

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 0.00 0.00 0.00 0 0

┌──[root@liruilongs.github.io]-[~]
└─$

top 综合情况监控

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
top - 01:46:43 up  5:29,  1 user,  load average: 0.01, 0.02, 0.05
Tasks: 151 total, 1 running, 150 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 4030172 total, 3227596 free, 205988 used, 596588 buff/cache
KiB Swap: 10485756 total, 10485756 free, 0 used. 3531292 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
844 etcd 20 0 10.308g 22996 11100 S 2.7 0.6 11:10.71 etcd
572 root 20 0 220312 5208 3508 S 0.3 0.1 0:06.03 rsyslogd
592 root 20 0 298868 6260 4916 S 0.3 0.2 1:32.71 vmtoolsd
1 root 20 0 125408 3836 2440 S 0.0 0.1 0:09.03 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.61 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:00.27 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 0:05.72 rcu_sched
10 root rt 0 0 0 0 S 0.0 0.0 0:01.25 watchdog/0
11 root rt 0 0 0 0 S 0.0 0.0 0:01.15 watchdog/1
12 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 0:00.54 ksoftirqd/1
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
17 root 20 0 0 0 0 S 0.0 0.0 0:00.05 kdevtmpfs
18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
19 root 20 0 0 0 0 S 0.0 0.0 0:00.03 khungtaskd
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd
22 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
23 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
  • load average: 0.01 不能超过 0.7
  • 需要除以 cpu核数 ;数字1

2、检测系统历史运行负载:sar

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system]
└─$ pwd
/usr/lib/systemd/system
┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system]
└─$ cat sysstat.service
# /usr/lib/systemd/system/sysstat.service
# (C) 2012 Peter Schiffer (pschiffe <at> redhat.com)
#
# sysstat-10.1.5 systemd unit file:
# Insert a dummy record in current daily data file.
# This indicates that the counters have restarted from 0.

[Unit]
Description=Resets System Activity Logs

[Service]
Type=oneshot
RemainAfterExit=yes
User=root
ExecStart=/usr/lib64/sa/sa1 --boot

[Install]
WantedBy=multi-user.target

┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system]
└─$ systemctl status sysstat.service
● sysstat.service - Resets System Activity Logs
Loaded: loaded (/usr/lib/systemd/system/sysstat.service; enabled; vendor preset: enabled)
Active: active (exited) since Wed 2021-10-13 01:53:41 CST; 1 weeks 3 days ago
Main PID: 584 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/sysstat.service

Oct 13 01:53:41 liruilongs.github.io systemd[1]: Starting Resets System Activity Logs...
Oct 13 01:53:41 liruilongs.github.io systemd[1]: Started Resets System Activity Logs.

红帽8没有,需要自己开启,红帽7有

1
2
3
4
5
┌──[root@liruilongs.github.io]-[/]
└─$ cd /var/log/sa
┌──[root@liruilongs.github.io]-[/var/log/sa]
└─$ ls
sa01 sa03 sa04 sa05 sa06 sa08 sa09 sa13 sa14 sa23 sa24 sa30 sar03

CPU 监控信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
┌──[root@liruilongs.github.io]-[/var/log/sa]
└─$ sar -f sa24 -p # CPU 监控信息
Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/24/2021 _x86_64_ (2 CPU)

08:14:46 AM LINUX RESTART

08:20:03 AM CPU %user %nice %system %iowait %steal %idle
09:40:01 AM all 1.44 0.00 1.72 0.21 0.00 96.64
09:50:01 AM all 0.80 0.00 1.51 0.01 0.00 97.69
10:00:01 AM all 0.81 0.00 1.56 0.00 0.00 97.63
10:10:01 AM all 0.74 0.00 1.40 0.02 0.00 97.83
10:20:01 AM all 0.65 0.00 1.26 0.00 0.00 98.08
10:30:01 AM all 0.78 0.02 1.50 0.04 0.00 97.67
10:40:01 AM all 3.24 0.00 5.65 0.01 0.00 91.11
Average: all 1.18 0.00 2.10 0.03 0.00 96.69

负载信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
┌──[root@liruilongs.github.io]-[/var/log/sa]
└─$ sar -f sa24 -q # 负载信息
Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/24/2021 _x86_64_ (2 CPU)

08:14:46 AM LINUX RESTART

08:20:03 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
09:40:01 AM 0 215 0.06 0.40 0.42 0
09:50:01 AM 0 215 0.41 0.21 0.28 0
10:00:01 AM 1 215 0.27 0.13 0.20 0
10:10:01 AM 1 216 0.03 0.08 0.14 0
10:20:01 AM 0 216 0.00 0.04 0.10 0
10:30:01 AM 1 217 0.00 0.03 0.06 0
10:40:01 AM 0 215 0.07 0.16 0.14 0
Average: 0 216 0.12 0.15 0.19 0

内存信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
┌──[root@liruilongs.github.io]-[/var/log/sa]
└─$ sar -f sa24 -r # 内存信息
Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/24/2021 _x86_64_ (2 CPU)

08:14:46 AM LINUX RESTART

08:20:03 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
09:40:01 AM 3472808 557364 13.83 2076 266132 832840 5.74 157400 244716 8
09:50:01 AM 3470960 559212 13.88 2076 266516 832840 5.74 158004 244652 4
10:00:01 AM 3466736 563436 13.98 2076 270432 832972 5.74 158272 248436 0
10:10:01 AM 3465552 564620 14.01 2076 270856 833480 5.74 159624 247772 0
10:20:01 AM 3465732 564440 14.01 2076 271172 833584 5.74 159720 248028 0
10:30:01 AM 3457092 573080 14.22 2076 272904 833772 5.74 161872 247732 0
10:40:01 AM 3458440 571732 14.19 2076 271864 833032 5.74 160824 247256 0
Average: 3465331 564841 14.02 2076 269982 833217 5.74 159388 246942 2

网卡信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
┌──[root@liruilongs.github.io]-[/var/log/sa]
└─$ sar -f sa24 -n DEV # 网卡信息
Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/24/2021 _x86_64_ (2 CPU)

08:14:46 AM LINUX RESTART

08:20:03 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
09:40:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:40:01 AM ens32 0.34 0.37 0.04 0.33 0.00 0.00 0.00
09:40:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00
09:40:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:50:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00
09:50:01 AM ens32 1.19 0.78 0.10 0.36 0.00 0.00 0.00
09:50:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00
09:50:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:00:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:00:01 AM ens32 0.20 0.13 0.02 0.01 0.00 0.00 0.00
10:00:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00
10:00:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:10:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:10:01 AM ens32 0.49 0.50 0.04 0.54 0.00 0.00 0.00
10:10:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00
10:10:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:20:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:20:01 AM ens32 0.17 0.11 0.01 0.01 0.00 0.00 0.00
10:20:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00
10:20:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:30:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:30:01 AM ens32 0.04 0.03 0.00 0.00 0.00 0.00 0.00
10:30:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00
10:30:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:40:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00
10:40:01 AM ens32 0.08 0.06 0.01 0.00 0.00 0.00 0.00
10:40:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00
10:40:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
Average: br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: ens32 0.36 0.28 0.03 0.17 0.00 0.00 0.00
Average: lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00
Average: docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
┌──[root@liruilongs.github.io]-[/var/log/sa]
└─$

3、通过 Performance Co-Pilot(PCP) 收集性能数据

安装相关包

1
2
3
4
┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system]
└─$ yum -y install pcp
┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system]
└─$ yum install pcp-gui -y

启动相关服务

1
2
3
4
5
6
7
┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system]
└─$ systemctl enable --now pmcd # 启动监控服务
Created symlink from /etc/systemd/system/multi-user.target.wants/pmcd.service to /usr/lib/systemd/system/pmcd.service.
┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system]
└─$ systemctl enable --now pmlogger # 启动存数据的数据库
Created symlink from /etc/systemd/system/multi-user.target.wants/pmlogger.service to /usr/lib/systemd/system/pmlogger.service.
┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system]

通过pmchart,pminfo,pmval读取系统当前及历史运行数据

查看所有的监控内容

1
2
3
4
5
6
7
8
9
┌──[root@liruilongs.github.io]-[~]  
└─$ pminfo
jbd2.njournals
jbd2.transaction.count
jbd2.transaction.requested
......
┌──[root@liruilongs.github.io]-[~]
└─$ pminfo | wc -l
2052

查看所有的监控参数详细信息

1
2
3
4
5
6
7
8
9
┌──[root@liruilongs.github.io]-[~]
└─$ pminfo -dt proc.nprocs #参数详细信息

proc.nprocs [instantaneous number of processes]
Data Type: 32-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff
Semantics: instant Units: none
┌──[root@liruilongs.github.io]-[~]
└─$ pminfo proc.nprocs #进程监控
proc.nprocs

监控进程信息 pmval proc.nprocs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
┌──[root@liruilongs.github.io]-[~]
└─$ pmval proc.nprocs #进程监控

metric: proc.nprocs
host: liruilongs.github.io
semantics: instantaneous value
units: none
samples: all
158
158
158
158
158
158
^C

监控剩余内存的量 pmval -t 2 mem.freemem

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
┌──[root@liruilongs.github.io]-[~]
└─$ pmval -t 2 mem.freemem # 剩余内存信息

metric: mem.freemem
host: liruilongs.github.io
semantics: instantaneous value
units: Kbyte
samples: all
3471612
3471636
3471636
3471636
3471636
^C
┌──[root@liruilongs.github.io]-[~]
└─$

查看监控日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
┌──[root@liruilongs.github.io]-[~]
└─$ pminfo | grep mem.freemem
mem.freemem
┌──[root@liruilongs.github.io]-[~]
└─$ cd /var/log/pcp
┌──[root@liruilongs.github.io]-[/var/log/pcp]
└─$ ls
NOTICES pmcd pmie pmlogger pmproxy
┌──[root@liruilongs.github.io]-[/var/log/pcp]
└─$ cd pmlogger/
┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger]
└─$ ls
liruilongs.github.io pmlogger_check.log.prev pmlogger_daily-K.log.prev pmlogger_daily.stamp
┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger]
└─$ cd liruilongs.github.io/
┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger/liruilongs.github.io]
└─$ ls
20211023.02.05.0.xz 20211023.02.30-00.0.xz 20211023.02.30.0.xz 20211024.08.16.0 Latest
20211023.02.05.index 20211023.02.30-00.index 20211023.02.30.index 20211024.08.16.index pmlogger.log
20211023.02.05.meta.xz 20211023.02.30-00.meta.xz 20211023.02.30.meta.xz 20211024.08.16.meta pmlogger.log.prior
┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger/liruilongs.github.io]
└─$ pmval -a 20211024.08.16.0 mem.freemem # 查看监控日志 每秒

metric: mem.freemem
archive: 20211024.08.16.0
host: liruilongs.github.io
start: Sun Oct 24 08:17:06 2021
end: Sun Oct 24 10:02:06 2021
semantics: instantaneous value
units: Kbyte
samples: 6301
interval: 1.00 sec
08:17:06.095 No values available
08:17:07.095 No values available
08:17:08.095 3651184
08:17:09.095 3651184
08:17:10.095 3651184
08:17:11.095 3651184
08:17:12.095 3651184
08:17:13.095 3651184
08:17:14.095 3651184
08:17:15.095 3651184
08:17:16.095 3651184
08:17:17.095 3651184
08:17:18.095 3651184
08:17:19.095 3651184
08:17:20.095 3651184
08:17:21.095 3651184
08:17:22.095 3651184
08:17:23.095 3651184
08:17:24.095 3651184
08:17:25.095 3651184
08:17:26.095 3651184
......

加时间粒度

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger/liruilongs.github.io]
└─$ pmval -a 20211024.08.16.0 -t 500 mem.freemem # 加时间粒度

metric: mem.freemem
archive: 20211024.08.16.0
host: liruilongs.github.io
start: Sun Oct 24 08:17:06 2021
end: Sun Oct 24 10:08:06 2021
semantics: instantaneous value
units: Kbyte
samples: 14
interval: 500.00 sec
08:17:06.095 No values available
08:25:26.095 3488128
08:33:46.095 3488128
08:42:06.095 3488128
08:50:26.095 3488128
08:58:46.095 3488128
09:07:06.095 3488128
09:15:26.095 3488128
09:23:46.095 3488128
09:32:06.095 3488128
09:40:26.095 3472784
09:48:46.095 3472024
09:57:06.095 3467052
10:05:26.095 3466276
┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger/liruilongs.github.io]
└─$

二、如何进行内核参数调优

/proc & /sys

/proc 存放内存中数据,每次重启就没了。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
┌──[root@liruilongs.github.io]-[/]
└─$ cd proc/
┌──[root@liruilongs.github.io]-[/proc]
└─$ ls
1 16836 241 269 284 3 313 362 45 647 853 crypto keys pagetypeinfo tty
10 16842 242 270 285 300 314 363 469 649 856 devices key-users partitions uptime
1041 16843 250 271 286 301 315 364 47 66 857 diskstats kmsg sched_debug version
1043 16850 251 272 287 302 316 365 5 6671 861 dma kpagecount schedstat vmallocinfo
1045 16862 253 273 288 303 317 366 506 7 864 driver kpageflags scsi vmstat
1050 17 254 274 289 304 318 367 509 7603 8922 execdomains loadavg self zoneinfo
1051 18 255 275 290 305 319 368 570 7604 9 fb locks slabinfo
1054 19 256 276 291 306 32 369 581 7606 98 filesystems mdstat softirqs
1055 2 262 277 292 307 320 370 583 8 acpi fs meminfo stat
1056 20 263 278 293 308 321 371 585 8095 buddyinfo interrupts misc swaps
11 21 264 279 294 309 322 372 594 8158 bus iomem modules sys
12 22 265 280 295 31 33 373 595 8159 cgroups ioports mounts sysrq-trigger
13 23 266 281 296 310 34 42 596 8160 cmdline irq mpt sysvipc
15 24 267 282 298 311 348 44 627 8161 consoles kallsyms mtrr timer_list
16006 240 268 283 299 312 361 441 642 8162 cpuinfo kcore net timer_stats
┌──[root@liruilongs.github.io]-[/proc]
└─$ pwd
/proc
┌──[root@liruilongs.github.io]-[/proc]
└─$ cd sys
┌──[root@liruilongs.github.io]-[/proc/sys]
└─$ ls
abi crypto debug dev fs kernel net user vm

内存调优目录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
┌──[root@liruilongs.github.io]-[/proc/sys]
└─$ cd vm # 内存调优目录
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ ls
admin_reserve_kbytes hugepages_treat_as_movable mmap_min_addr overcommit_memory
block_dump hugetlb_shm_group mmap_rnd_bits overcommit_ratio
compact_memory laptop_mode mmap_rnd_compat_bits page-cluster
dirty_background_bytes legacy_va_layout nr_hugepages panic_on_oom
dirty_background_ratio lowmem_reserve_ratio nr_hugepages_mempolicy percpu_pagelist_fraction
dirty_bytes max_map_count nr_overcommit_hugepages stat_interval
dirty_expire_centisecs memory_failure_early_kill nr_pdflush_threads swappiness
dirty_ratio memory_failure_recovery numa_zonelist_order user_reserve_kbytes
dirty_writeback_centisecs min_free_kbytes oom_dump_tasks vfs_cache_pressure
drop_caches min_slab_ratio oom_kill_allocating_task zone_reclaim_mode
extfrag_threshold min_unmapped_ratio overcommit_kbytes
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$

查看内核参数:cat & sysctl -a

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ sysctl -a # 查看所有调优参数
abi.vsyscall32 = 1
crypto.fips_enabled = 0
debug.exception-trace = 1
debug.kprobes-optimization = 1
debug.panic_on_rcu_stall = 0
dev.hpet.max-user-freq = 64
dev.mac_hid.mouse_button2_keycode = 97
dev.mac_hid.mouse_button3_keycode = 100
dev.mac_hid.mouse_button_emulation = 0
dev.parport.default.spintime = 500
dev.parport.default.timeslice = 200
dev.raid.speed_limit_max = 200000
dev.raid.speed_limit_min = 1000
dev.scsi.logging_level = 0
fs.aio-max-nr = 65536
fs.aio-nr = 0
..............

根据变量找对应文件参数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ sysctl -a | grep net.ipv4.ip_forward
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_use_pmtu = 0
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.br-4b3da203747c.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.docker0.stable_secret"
sysctl: reading key "net.ipv6.conf.ens32.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ cd ../net/ipv
ipv4/ ipv6/
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ cd ../net/ipv4/
┌──[root@liruilongs.github.io]-[/proc/sys/net/ipv4]
└─$ cat ip_forward
1

调整内核参数:echo & sysctl

设置调优参数:临时调整

1
2
3
4
5
6
7
8
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ cat swappiness
30
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ echo 40 > swappiness ## 临时调整
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ cat swappiness
40

设置调优参数:永久调整

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ cat /etc/sysc
sysconfig/ sysctl.conf sysctl.d/
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv4.ip_forward = 1
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ echo "vm.swappiness = 20" >> /etc/sysctl.conf ## 永久调整
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv4.ip_forward = 1
vm.swappiness = 20
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ sysctl -p
net.ipv4.ip_forward = 1
vm.swappiness = 20
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$

内核调优之清理缓存

如何使用帮助调整内核参数

1
2
3
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ man -K drop_caches
--Man-- next: proc(5) [ view (return) | skip (Ctrl-D) | quit (Ctrl-C) ]
清理缓存
在这里插入图片描述
在这里插入图片描述
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ cat drop_caches #缓存处理
0
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ man -K drop_caches
No manual entry for drop_caches
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ man -K drop_caches
No manual entry for drop_caches
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ man -K ip_forward
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ free -m
total used free shared buff/cache available
Mem: 3935 212 3357 16 366 3440
Swap: 10239 0 10239
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ echo 3 > /proc/sys/vm/drop_caches
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ free -m
total used free shared buff/cache available
Mem: 3935 200 3575 16 159 3504
Swap: 10239 0 10239
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$

调整内核模块参数:lsmod、modinfo、modprobe

加载的内核模块列表 lsmod

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ lsmod # 大小 调用次数
Module Size Used by
binfmt_misc 17468 1
xt_conntrack 12760 2
ipt_MASQUERADE 12678 2
nf_nat_masquerade_ipv4 13412 1 ipt_MASQUERADE
nf_conntrack_netlink 40449 0
nfnetlink 14696 2 nf_conntrack_netlink
xt_addrtype 12676 2
iptable_filter 12810 1
iptable_nat 12875 1
nf_conntrack_ipv4 15053 3
nf_defrag_ipv4 12729 1 nf_conntrack_ipv4
。。。。。
。。。。。。
crct10dif_common 12595 3 crct10dif_pclmul,crct10dif_generic,crc_t10dif
crc32c_intel 22079 1
ahci 34042 0
drm 370825 2 ttm,drm_kms_helper
libahci 31992 1 ahci
mptspi 22542 2
ata_piix 35038 0
scsi_transport_spi 30732 1 mptspi
mptscsih 40150 1 mptspi
libata 238896 5 ahci,pata_acpi,libahci,ata_generic,ata_piix
e1000 137500 0
serio_raw 13413 0
mptbase 105960 2 mptspi,mptscsih
i2c_core 40756 3 drm,i2c_piix4,drm_kms_helper
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$

查看内核模块参数 modinfo

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ modinfo kvm # 查看内核模块参数
filename: /lib/modules/3.10.0-693.el7.x86_64/kernel/arch/x86/kvm/kvm.ko.xz
license: GPL
author: Qumranet
rhelversion: 7.4
srcversion: FA3AAB0FB1DD5C7B9D69811
depends: irqbypass
intree: Y
vermagic: 3.10.0-693.el7.x86_64 SMP mod_unload modversions
signer: CentOS Linux kernel signing key
sig_key: DA:18:7D:CA:7D:BE:53:AB:05:BD:13:BD:0C:4E:21:F4:22:B6:A4:9C
sig_hashalgo: sha256
parm: ignore_msrs:bool
parm: min_timer_period_us:uint
parm: kvmclock_periodic_sync:bool
parm: tsc_tolerance_ppm:uint
parm: lapic_timer_advance_ns:uint
parm: vector_hashing:bool
parm: halt_poll_ns:uint
parm: halt_poll_ns_grow:uint
parm: halt_poll_ns_shrink:uint
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$

内核模块调优之嵌套虚拟化设置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ modinfo kvm
filename: /lib/modules/3.10.0-693.el7.x86_64/kernel/arch/x86/kvm/kvm.ko.xz
license: GPL
author: Qumranet
rhelversion: 7.4
srcversion: FA3AAB0FB1DD5C7B9D69811
depends: irqbypass
intree: Y
vermagic: 3.10.0-693.el7.x86_64 SMP mod_unload modversions
signer: CentOS Linux kernel signing key
sig_key: DA:18:7D:CA:7D:BE:53:AB:05:BD:13:BD:0C:4E:21:F4:22:B6:A4:9C
sig_hashalgo: sha256
parm: ignore_msrs:bool
parm: min_timer_period_us:uint
parm: kvmclock_periodic_sync:bool
parm: tsc_tolerance_ppm:uint
parm: lapic_timer_advance_ns:uint
parm: vector_hashing:bool
parm: halt_poll_ns:uint
parm: halt_poll_ns_grow:uint
parm: halt_poll_ns_shrink:uint
┌──[root@liruilongs.github.io]-[/proc/sys/vm]
└─$ modinfo kvm | grep ignore_msrs # 设置允许嵌套虚拟化
parm: ignore_msrs:bool
┌──[root@liruilongs.github.io]-[/etc/modprobe.d]
└─$ echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf ##从起机器

三、如何限制服务的资源占用

资源消耗分为用户和服务,消耗:CPU,内存和 I/O

经典的限制通过 pam_limit 模块配合 /etc/security/limits.conf 实现(用户)

用户登录会加载pam_limit 模块,pam_limit 模块读取配置文件 /etc/security/limits.conf限制用户资源的占用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
┌──[root@liruilongs.github.io]-[/etc/modprobe.d]
└─$ cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#This file sets the resource limits for the users logged in via PAM.
#It does not affect resource limits of the system services.
#
#Also note that configuration files in /etc/security/limits.d directory,
#which are read in alphabetical order, override the settings in this
#file in case the domain is the same or more specific.
#That means for example that setting a limit for wildcard domain here
#can be overriden with a wildcard setting in a config file in the
#subdirectory, but a user specific setting here can be overriden only
#with a user specific setting in the subdirectory.
#
#Each line describes a limit for a user in the form:
#
#<domain> <type> <item> <value>
#
#Where:
#<domain> can be:
# - a user name
# - a group name, with @group syntax
# - the wildcard *, for default entry
# - the wildcard %, can be also used with %group syntax,
# for maxlogin limit
#
#<type> can have the two values:
# - "soft" for enforcing the soft limits
# - "hard" for enforcing hard limits
#
#<item> can be one of the following:
# - core - limits the core file size (KB) 核心文件大小
# - data - max data size (KB)
# - fsize - maximum filesize (KB)
# - memlock - max locked-in-memory address space (KB)
# - nofile - max number of open file descriptors # 访问文件数量
# - rss - max resident set size (KB)
# - stack - max stack size (KB)
# - cpu - max CPU time (MIN) # cup时间
# - nproc - max number of processes
# - as - address space limit (KB)
# - maxlogins - max number of logins for this user #最多的登录数
# - maxsyslogins - max number of logins on the system
# - priority - the priority to run user process with
# - locks - max number of file locks the user can hold
# - sigpending - max number of pending signals
# - msgqueue - max memory used by POSIX message queues (bytes)
# - nice - max nice priority allowed to raise to values: [-20, 19]
# - rtprio - max realtime priority
#
#<domain> <type> <item> <value>
#

#* soft core 0
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#@student - maxlogins 4

# End of file

限制服务的资源之限制用户通过ssh的登录数

1
2
3
┌──[root@liruilongs.github.io]-[/etc/modprobe.d]
└─$vim /etc/security/limits.conf
student hard maxlogins 4

也可以通过 cgroup 实现资源的限制(服务)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
┌──[root@liruilongs.github.io]-[/etc/modprobe.d]
└─$ md5sum /dev/urandom & #启动一个进程 cpu100%
[1] 38162
┌──[root@liruilongs.github.io]-[/etc/modprobe.d]
└─$ top
top - 13:37:42 up 4:10, 1 user, load average: 1.03, 0.80, 0.43
Tasks: 158 total, 2 running, 156 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.0 us, 53.9 sy, 0.0 ni, 43.3 id, 0.0 wa, 0.0 hi, 0.8 si, 0.0 st
KiB Mem : 4030172 total, 3637840 free, 207252 used, 185080 buff/cache
KiB Swap: 10485756 total, 10485756 free, 0 used. 3575580 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
861 etcd 20 0 10.254g 22572 11092 S 9.7 0.6 22:07.49 etcd
9 root 20 0 0 0 0 S 1.7 0.0 0:28.00 rcu_sched
595 root 20 0 298868 6260 4916 S 1.6 0.2 2:27.63 vmtoolsd
371 root 20 0 0 0 0 S 0.4 0.0 1:23.11 xfsaild/sda1
1 root 20 0 51688 3960 2624 S 0.4 0.1 0:31.08 systemd
36765 root 20 0 0 0 0 S 0.2 0.0 0:03.13 kworker/1:0
583 root 20 0 21616 1284 972 S 0.2 0.0 0:09.59 irqbalance
857 root 20 0 573468 16760 6020 S 0.2 0.4 0:14.24 tuned
13 root 20 0 0 0 0 S 0.2 0.0 0:06.16 ksoftirqd/1
864 root 20 0 1015116 42652 13740 S 0.1 1.1 0:22.33 containerd
853 root 20 0 228220 5128 3240 S 0.1 0.1 0:09.13 httpd
594 root 20 0 224504 12424 3348 S 0.1 0.3 0:08.61 rsyslogd
649 chrony 20 0 117704 1772 1312 S 0.1 0.0 0:03.47 chronyd
7604 root 20 0 151984 5352 4088 R 0.1 0.1 0:05.89 sshd
38166 root 20 0 161908 2264 1556 R 0.1 0.1 0:00.10 top
1055 root 20 0 81744 3132 2168 S 0.0 0.1 0:10.09 pmdalinux
8095 root 20 0 0 0 0 S 0.0 0.0 0:01.54 kworker/u256:0
11 root rt 0 0 0 0 S 0.0 0.0 0:02.17 watchdog/1
1051 root 20 0 81628 2700 1912 S 0.0 0.1 0:03.07 pmdaproc
2 root 20 0 0 0 0 S 0.0 0.0 0:00.09 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:01.56 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:01.08 migration/0
[1]+ Terminated md5sum /dev/urandom
┌──[root@liruilongs.github.io]-[/etc/modprobe.d]
└─$

在这里插入图片描述

现在我们把这个写成一个服务,然后通过cgroup做简单限制。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
┌──[root@liruilongs.github.io]-[~]
└─$ vim /etc/systemd/system/md5sum.service
┌──[root@liruilongs.github.io]-[~]
└─$ cat /etc/systemd/system/md5sum.service
[Unit]
Description=MD5 Demo

[Service]
CPUQuota=40%
ExecStart=/usr/bin/md5sum /dev/urandom

[Install]
WantedBy=multi-user.target
┌──[root@liruilongs.github.io]-[~]
└─$
┌──[root@liruilongs.github.io]-[~]
└─$ systemctl daemon-reload
┌──[root@liruilongs.github.io]-[~]
└─$ systemctl start md5sum.service
┌──[root@liruilongs.github.io]-[~]
└─$ top
top - 19:12:14 up 9:44, 1 user, load average: 0.29, 0.20, 0.11
Tasks: 159 total, 2 running, 157 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.5 us, 20.3 sy, 0.0 ni, 78.8 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 4030172 total, 3606960 free, 209832 used, 213380 buff/cache
KiB Swap: 10485756 total, 10485756 free, 0 used. 3558764 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
59598 root 20 0 107920 612 516 R 39.7 0.0 0:02.54 md5sum ##CPU限制为40%
861 etcd 20 0 10.254g 22572 11092 S 6.3 0.6 51:24.88 etcd
59601 root 20 0 161908 2268 1560 R 1.0 0.1 0:00.08 top
595 root 20 0 298868 6260 4916 S 0.7 0.2 5:35.80 vmtoolsd
1 root 20 0 51688 4000 2648 S 0.3 0.1 1:15.99 systemd
9 root 20 0 0 0 0 S 0.3 0.0 0:50.90 rcu_sched
371 root 20 0 0 0 0 S 0.3 0.0 3:02.41 xfsaild/sda1
6671 root 20 0 1309148 66444 25620 S 0.3 1.6 0:48.35 dockerd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.13 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:04.25 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:02.39 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
10 root rt 0 0 0 0 S 0.0 0.0 0:03.13 watchdog/0
11 root rt 0 0 0 0 S 0.0 0.0 0:04.80 watchdog/1
12 root rt 0 0 0 0 S 0.0 0.0 0:02.72 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 0:11.18 ksoftirqd/1
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
17 root 20 0 0 0 0 S 0.0 0.0 0:00.11 kdevtmpfs
18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
19 root 20 0 0 0 0 S 0.0 0.0 0:00.08 khungtaskd
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd
┌──[root@liruilongs.github.io]-[~]
└─$

这个具体小伙伴可以看看这篇博客: https://blog.csdn.net/sanhewuyang/article/details/120735766

关于cgroup参数设置可以使用帮助文档

1
2
3
4
5
6
┌──[root@liruilongs.github.io]-[~]
└─$ man -k systemd
┌──[root@liruilongs.github.io]-[~]
└─$ man systemd.resource-control
┌──[root@liruilongs.github.io]-[~]
└─$

为 systemd unit 实现资源限制

这个老师没讲,以后研究下,应该也和cgroup有关

自定义tuned调优配置集

tuned 实现系统傻瓜化和集成化调优,操作系统预装多个调优场景配置集,这是一道RHCAS的考试题

查看调优策略

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
┌──[root@liruilongs.github.io]-[~]
└─$ tuned-adm list
Available profiles:
- balanced - General non-specialized tuned profile
- desktop - Optimize for the desktop use-case
- latency-performance - Optimize for deterministic performance at the cost of increased power consumption
- network-latency - Optimize for deterministic performance at the cost of increased power consumption, focused on low latency network performance
- network-throughput - Optimize for streaming network throughput, generally only necessary on older CPUs or 40G+ networks
- powersave - Optimize for low power consumption
- throughput-performance - Broadly applicable tuning that provides excellent performance across a variety of common server workloads
- virtual-guest - Optimize for running inside a virtual guest
- virtual-host - Optimize for running KVM guests
Current active profile: virtual-guest
┌──[root@liruilongs.github.io]-[~]
└─$ tuned-adm recommend # 查看推荐的策略
virtual-guest
┌──[root@liruilongs.github.io]-[~]
└─$ tuned-adm profile virtual-guest

调优策略位置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
┌──[root@liruilongs.github.io]-[~]
└─$ cd /usr/lib/tuned/;ls ## 调优参数
balanced latency-performance powersave virtual-guest
desktop network-latency recommend.conf virtual-host
functions network-throughput throughput-performance
┌──[root@liruilongs.github.io]-[/usr/lib/tuned]
└─$ cd virtual-guest/
┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest]
└─$ ls
tuned.conf
┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest]
└─$ cat tuned.conf
#
# tuned configuration
#

[main]
summary=Optimize for running inside a virtual guest
include=throughput-performance

[sysctl]
# If a workload mostly uses anonymous memory and it hits this limit, the entire
# working set is buffered for I/O, and any more write buffering would require
# swapping, so it's time to throttle writes until I/O can catch up. Workloads
# that mostly use file mappings may be able to use even higher values.
#
# The generator of dirty data starts writeback at this percentage (system default
# is 20%)
vm.dirty_ratio = 30

# Filesystem I/O is usually much more efficient than swapping, so try to keep
# swapping low. It's usually safe to go even lower than this on systems with
# server-grade storage.
vm.swappiness = 30
┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest]
└─$

在/etc/tuned目录中编辑自定义tuned profile,配置集中的条目可以相互引用,并通过各种plugin自定义
获取 yum仓库的tuned profile

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest]
└─$ yum list | grep tuned
tuned.noarch 2.8.0-5.el7 @anaconda
tuned.noarch 2.11.0-11.el7_9 updates
tuned-gtk.noarch 2.11.0-11.el7_9 updates
tuned-profiles-atomic.noarch 2.11.0-11.el7_9 updates
tuned-profiles-compat.noarch 2.11.0-11.el7_9 updates
tuned-profiles-cpu-partitioning.noarch 2.11.0-11.el7_9 updates
tuned-profiles-mssql.noarch 2.11.0-11.el7_9 updates
tuned-profiles-oracle.noarch 2.11.0-11.el7_9 updates
tuned-utils.noarch 2.11.0-11.el7_9 updates
tuned-utils-systemtap.noarch 2.11.0-11.el7_9 updates
┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest]
└─$ yum -y install tuned-profiles-oracle.noarch
....
┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest]
└─$ tuned-adm list
Available profiles:
- balanced - General non-specialized tuned profile
- desktop - Optimize for the desktop use-case
- hpc-compute - Optimize for HPC compute workloads
- latency-performance - Optimize for deterministic performance at the cost of increased power consumption
- network-latency - Optimize for deterministic performance at the cost of increased power consumption, focused on low latency network performance
- network-throughput - Optimize for streaming network throughput, generally only necessary on older CPUs or 40G+ networks
- oracle - Optimize for Oracle RDBMS
- powersave - Optimize for low power consumption
- throughput-performance - Broadly applicable tuning that provides excellent performance across a variety of common server workloads
- virtual-guest - Optimize for running inside a virtual guest
- virtual-host - Optimize for running KVM guests
Current active profile: virtual-guest
┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest]
└─$

四、通过SystemTap配置内核模块进行系统底层分析:

嗯,这部分有些复杂,有些包需要订阅,先记录下,以后有机会学习

通过SystemTap配置内核模块进行系统底层分析:
SystemTap 可以简易的探索测量 kernel 中任意部件
Kernel开发者通过kprobe在kernel功能前后加入测试代码
SystemTap需要gcc,kernel-debuginfo,kernel-devel软件支持
stap命令将*.stp脚本文件编译为kernel模块,通过staprun命令运行
发布于

2021-10-23

更新于

2024-01-14

许可协议

评论
加载中,最新评论有1分钟缓存...
Your browser is out-of-date!

Update your browser to view this website correctly.&npsb;Update my browser now

×